feat: preserve field ids when creating table metadata - #902
Open
WZhuo wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds a new table-metadata construction path that preserves schema field IDs, validating ID correctness/uniqueness while still assigning partition field IDs via new-table rules.
Changes:
- Introduces
TableMetadata::MakeWithFieldIdsAPI for creating new table metadata while keeping schema field IDs. - Adds schema field-ID validation (positive + globally unique across nested fields).
- Extends unit tests to cover preserved IDs and rejection of invalid IDs.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| src/iceberg/test/table_metadata_builder_test.cc | Adds tests for MakeWithFieldIds preserving schema IDs and rejecting invalid IDs. |
| src/iceberg/table_metadata.h | Declares the new MakeWithFieldIds public API and documents its behavior. |
| src/iceberg/table_metadata.cc | Implements MakeWithFieldIds and adds preserved field-ID validation logic. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| return {}; | ||
| }; | ||
|
|
||
| return validate_type(schema); |
Comment on lines
+265
to
+274
| Result<std::unique_ptr<TableMetadata>> TableMetadata::MakeWithFieldIds( | ||
| const iceberg::Schema& schema, const iceberg::PartitionSpec& spec, | ||
| const iceberg::SortOrder& sort_order, const std::string& location, | ||
| const std::unordered_map<std::string, std::string>& properties, int format_version) { | ||
| for (const auto& [key, _] : properties) { | ||
| if (TableProperties::reserved_properties().contains(key)) { | ||
| return InvalidArgument( | ||
| "Table properties should not contain reserved properties, but got {}", key); | ||
| } | ||
| } |
|
|
||
| Status ValidatePreservedFieldIds(const Schema& schema) { | ||
| std::unordered_set<int32_t> field_ids; | ||
| std::function<Status(const Type&)> validate_type = [&](const Type& type) -> Status { |
Comment on lines
+128
to
+147
| std::function<Status(const Type&)> validate_type = [&](const Type& type) -> Status { | ||
| if (!type.is_nested()) { | ||
| return {}; | ||
| } | ||
|
|
||
| const auto& nested = internal::checked_cast<const NestedType&>(type); | ||
| for (const auto& field : nested.fields()) { | ||
| if (field.field_id() <= Schema::kInitialColumnId) { | ||
| return InvalidSchema("Invalid field id {} for '{}'", field.field_id(), | ||
| field.name()); | ||
| } | ||
| if (!field_ids.insert(field.field_id()).second) { | ||
| return InvalidSchema("Duplicate field id found: {}", field.field_id()); | ||
| } | ||
| ICEBERG_RETURN_UNEXPECTED(validate_type(*field.type())); | ||
| } | ||
| return {}; | ||
| }; | ||
|
|
||
| return validate_type(schema); |
| return SortOrder::Make(order_id, std::move(sort_fields)); | ||
| } | ||
|
|
||
| Status ValidatePreservedFieldIds(const Schema& schema) { |
Member
|
I think we've already supported this feature like the code below so perhaps this PR is redundant? ICEBERG_ASSIGN_OR_RAISE(auto last_id, schema->HighestFieldId());
auto builder = TableMetadataBuilder::BuildFromEmpty(2);
builder->AssignUUID()
.SetLocation(location)
.SetCurrentSchema(schema, last_id)
.SetDefaultPartitionSpec(spec)
.SetDefaultSortOrder(order)
.SetProperties(properties);
ICEBERG_ASSIGN_OR_RAISE(auto metadata, builder->Build()); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TableMetadata::MakeWithFieldIdsfor preserving schema field IDsTests
cmake --build build --target table_test --parallel 4build/src/iceberg/test/table_test --gtest_filter='TableMetadataTest.MakeWithFieldIds*'